[Ingest Manager] Support updated package output structure #69864

jen-huang · 2020-06-24T23:06:29Z

‼️ Do not merge, PR needs to be merged in tandem with elastic/package-registry#514

Summary

Part of elastic/beats#19082. This PR makes the necessary changes for Ingest Manager to support package registry changes done in elastic/package-registry#514:

Update EPM's registry types and schemas to reflect registry package output changes
- dataset.id was renamed to dataset.name, PR updates all references
Update data source saved object mappings to change stream[].dataset: string; to stream[].dataset: { name: string; type: string}
- Associated types and schemas were also updated
Inline agent templates were removed from package registry in favor of template_path, datasourceService.assignPackageStream() and related methods were updated to fetch the stream templates
Moved inputs[].package information in agent config yaml to inputs[].meta.package
Updated associated tests for everything above

Example updated agent config yaml

Expand me (yaml for default config with system integration)

id: c455d4d0-b71f-11ea-a989-ef5894976394
revision: 2
settings:
  monitoring:
    enabled: true
    use_output: default
    logs: true
    metrics: true
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://localhost:9200'
inputs:
  - id: c8a076d0-b71f-11ea-a989-ef5894976394
    name: system-1
    type: logs
    use_output: default
    meta:
      package:
        name: system
        version: 0.3.0
    dataset:
      namespace: default
    streams:
      - id: logs-system.auth
        dataset:
          name: system.auth
          type: logs
        paths:
          - /var/log/auth.log*
          - /var/log/secure*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.5.0
      - id: logs-system.syslog
        dataset:
          name: system.syslog
          type: logs
        paths:
          - /var/log/messages*
          - /var/log/syslog*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              target: ''
              fields:
                ecs.version: 1.5.0
  - id: c8a076d0-b71f-11ea-a989-ef5894976394
    name: system-1
    type: system/metrics
    use_output: default
    meta:
      package:
        name: system
        version: 0.3.0
    dataset:
      namespace: default
    streams:
      - id: system/metrics-system.cpu
        dataset:
          name: system.cpu
          type: metrics
        metricsets:
          - cpu
        core.metrics: null
        cpu.metrics:
          - percentages
          - normalized_percentages
        period: 10s
      - id: system/metrics-system.diskio
        dataset:
          name: system.diskio
          type: metrics
        metricsets:
          - diskio
      - id: system/metrics-system.load
        dataset:
          name: system.load
          type: metrics
        metricsets:
          - load
        core.metrics: null
        cpu.metrics: null
        period: 10s
        process.include_top_n.by_cpu: null
        process.include_top_n.by_memory: null
        processes: null
      - id: system/metrics-system.memory
        dataset:
          name: system.memory
          type: metrics
        metricsets:
          - memory
        core.metrics: null
        cpu.metrics: null
        period: 10s
        process.include_top_n.by_cpu: null
        process.include_top_n.by_memory: null
        processes: null
      - id: system/metrics-system.network
        dataset:
          name: system.network
          type: metrics
        metricsets:
          - network
        core.metrics: null
        cpu.metrics: null
        period: 10s
        process.include_top_n.by_cpu: null
        process.include_top_n.by_memory: null
        processes: null
      - id: system/metrics-system.process
        dataset:
          name: system.process
          type: metrics
        metricsets:
          - process
        period: 10s
        process.include_top_n.by_cpu: 5
        process.include_top_n.by_memory: 5
        processes:
          - .*
      - id: system/metrics-system.process_summary
        dataset:
          name: system.process_summary
          type: metrics
        metricsets:
          - process_summary
        core.metrics: null
        cpu.metrics: null
        period: 10s
        process.include_top_n.by_cpu: null
        process.include_top_n.by_memory: null
        processes: null
      - id: system/metrics-system.socket_summary
        dataset:
          name: system.socket_summary
          type: metrics
        metricsets:
          - socket_summary
        core.metrics: null
        cpu.metrics: null
        period: 10s
        process.include_top_n.by_cpu: null
        process.include_top_n.by_memory: null
        processes: null
      - id: system/metrics-system.uptime
        dataset:
          name: system.uptime
          type: metrics
        metricsets:
          - uptime
        core.metrics: null
        cpu.metrics: null
        period: 10s
        processes: null

Testing

Use:

Local build of package registry from Update package output structure package-registry#514 with xpack.ingestManager.epm.registryUrl: 'http://localhost:8080'
Latest agent snapshot
Fresh ES instance with no previous data

All functionality should work normally with no UI changes, apart from slight agent config YAML view changes 😄

…istry-datasources

…pping to be object containing `name` with instead of just a string

…ng it per stream

…istry-datasources

elasticmachine · 2020-06-25T21:26:24Z

Pinging @elastic/ingest-management (Team:Ingest Management)

paul-tavares · 2020-06-26T13:14:54Z

I will try to check this out locally today to see if its a matter of just updating the test case to match the new full agent config output

…istry-datasources

jen-huang · 2020-06-26T16:00:39Z

@paul-tavares The test failed because I moved package info under a new field, meta, so the snapshot needs to be updated. Working on it now!

paul-tavares · 2020-06-26T17:44:07Z

@jen-huang cool. I assume all that moving around is being coordinated with Agent and that the Endpoint specific data will still reach the endpoint, so 👍

ruflin · 2020-06-29T12:14:01Z

x-pack/plugins/ingest_manager/common/services/config_to_yaml.ts

@@ -16,7 +16,7 @@ const CONFIG_KEYS_ORDER = [
  'inputs',
  'enabled',
  'use_output',
-  'package',
+  'meta',


@michalpristas for your awareness. As Agent ignores this at the moment, it should not have any effect.

ruflin · 2020-06-29T12:20:39Z

@elasticmachine merge upstream

ruflin

@jen-huang I tested this PR together with the registry and the newest agent and everything seems to work as expected. I suggest we get the two PR's in in parallel and open quickly a backport for KB.

kibanamachine · 2020-06-29T13:58:31Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: e06bf3f

Build metrics

✅ unchanged

History

💚 Build #56744 succeeded cd6f2d1
💔 Build #56505 failed 3b4e75e
💚 Build #56129 succeeded f34e642

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…70263) * Update EPM package registry typings to reflect registry changes * Change `dataset.id` references to `dataset.name` * Fix RegistryStream * Fix packageToConfigDatasourceInputs service * Fix assignPackageStream service * Fix validateDatasource service * Fix configure data source components * Fix variable * Fix stream template installation * Add support for `input[].dataset.type` and change `stream.dataset` mapping to be object containing `name` with instead of just a string * Nest package information under `meta` in agent config yaml * Move `dataset.type` to stream level instead of input level * Make single call to fetch registry package information instead of doing it per stream * Fix type issues * Update endpoint test assertion Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

…ata-streams * 'master' of github.com:elastic/kibana: (50 commits) [Logs UI] [Alerting] "Group by" functionality (elastic#68250) [Discover] Deangularize Skip to bottom button (elastic#69811) Implement recursive plugin discovery (elastic#68811) Use ts-expect-error in platform code (elastic#69883) [SIEM][Detection Engine][Lists] Moves getQueryFilter to common folder for use by both front and backend [Ingest Manager][SECURITY SOLUTION] adjust config reassign link and add roundtrip to Reassignment flow (elastic#70208) [Security][Lists] Add API functions and react hooks for value list APIs (elastic#69603) [ILM] Fix bug when clearing priority field (elastic#70154) [Platform][Security] Updates cluster_manager ignorePaths to include security scripts (elastic#70139) [IngestManager] Allow to filter agent by packages (elastic#69731) [code coverage] exclude folders: test_helpers, tests_bundle (elastic#70199) [Metrics UI] UX improvements for saved views (elastic#69910) [APM] docs: unique transaction troubleshooting (elastic#69831) Cross cluster search functional test with minimun privileges assigned to the test_user (elastic#70007) [Maps] choropleth layer wizard (elastic#69699) Make custom errors by extending Error (elastic#69966) [Ingest Manager] Support updated package output structure (elastic#69864) Resolver test coverage (elastic#70246) Async Discover search test (elastic#64388) [ui-shared-deps] include styled-components (elastic#69322) ... # Conflicts: # x-pack/plugins/snapshot_restore/server/types.ts

…bana into alerting/consumer-based-rbac * 'alerting/consumer-based-rbac' of github.com:gmmorris/kibana: (49 commits) [Discover] Deangularize Skip to bottom button (elastic#69811) Implement recursive plugin discovery (elastic#68811) Use ts-expect-error in platform code (elastic#69883) [SIEM][Detection Engine][Lists] Moves getQueryFilter to common folder for use by both front and backend [Ingest Manager][SECURITY SOLUTION] adjust config reassign link and add roundtrip to Reassignment flow (elastic#70208) [Security][Lists] Add API functions and react hooks for value list APIs (elastic#69603) [ILM] Fix bug when clearing priority field (elastic#70154) [Platform][Security] Updates cluster_manager ignorePaths to include security scripts (elastic#70139) [IngestManager] Allow to filter agent by packages (elastic#69731) [code coverage] exclude folders: test_helpers, tests_bundle (elastic#70199) [Metrics UI] UX improvements for saved views (elastic#69910) [APM] docs: unique transaction troubleshooting (elastic#69831) Cross cluster search functional test with minimun privileges assigned to the test_user (elastic#70007) [Maps] choropleth layer wizard (elastic#69699) Make custom errors by extending Error (elastic#69966) [Ingest Manager] Support updated package output structure (elastic#69864) Resolver test coverage (elastic#70246) Async Discover search test (elastic#64388) [ui-shared-deps] include styled-components (elastic#69322) SECURITY-ENDPOINT: add host properties (elastic#70238) ...

) * Update EPM package registry typings to reflect registry changes * Change `dataset.id` references to `dataset.name` * Fix RegistryStream * Fix packageToConfigDatasourceInputs service * Fix assignPackageStream service * Fix validateDatasource service * Fix configure data source components * Fix variable * Fix stream template installation * Add support for `input[].dataset.type` and change `stream.dataset` mapping to be object containing `name` with instead of just a string * Nest package information under `meta` in agent config yaml * Move `dataset.type` to stream level instead of input level * Make single call to fetch registry package information instead of doing it per stream * Fix type issues * Update endpoint test assertion Co-authored-by: Elastic Machine <[email protected]>

jen-huang added 11 commits June 23, 2020 15:47

Update EPM package registry typings to reflect registry changes

93aeaa4

Change dataset.id references to dataset.name

cf240c3

Fix RegistryStream

93b83df

Fix packageToConfigDatasourceInputs service

eb77efc

Fix assignPackageStream service

cfdc57f

Fix validateDatasource service

5a2d688

Fix configure data source components

c3c75c4

Fix variable

a5a8f11

Merge remote-tracking branch 'upstream/master' into ingest/remove-reg…

7a63526

…istry-datasources

Fix stream template installation

fe5a6ae

Add support for input[].dataset.type and change stream.dataset ma…

f34e642

…pping to be object containing `name` with instead of just a string

jen-huang self-assigned this Jun 24, 2020

jen-huang added 5 commits June 25, 2020 12:58

Nest package information under meta in agent config yaml

7a01fd0

Move dataset.type to stream level instead of input level

67f7c89

Make single call to fetch registry package information instead of doi…

b9f0ddb

…ng it per stream

Fix type issues

2e0eaf0

Merge remote-tracking branch 'upstream/master' into ingest/remove-reg…

3b4e75e

…istry-datasources

jen-huang added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.9.0 v8.0.0 labels Jun 25, 2020

jen-huang marked this pull request as ready for review June 25, 2020 21:26

jen-huang requested a review from a team June 25, 2020 21:26

jen-huang requested a review from ruflin June 25, 2020 21:32

jen-huang changed the title ~~[WIP][Ingest Manager] Support updated package output structure~~ [Ingest Manager] Support updated package output structure Jun 25, 2020

ruflin mentioned this pull request Jun 25, 2020

[Elastic Agent] Change structure of Elastic Agent across the Stack elastic/beats#19082

Closed

15 tasks

jen-huang mentioned this pull request Jun 25, 2020

[Ingest Manager] Rename and restructure data source saved object #70018

Closed

9 tasks

Merge remote-tracking branch 'upstream/master' into ingest/remove-reg…

bf92368

…istry-datasources

Update endpoint test assertion

cd6f2d1

jen-huang requested review from a team as code owners June 26, 2020 16:04

ruflin reviewed Jun 29, 2020

View reviewed changes

Merge branch 'master' into ingest/remove-registry-datasources

e06bf3f

ruflin approved these changes Jun 29, 2020

View reviewed changes

jen-huang merged commit d9fcc58 into elastic:master Jun 29, 2020

jen-huang deleted the ingest/remove-registry-datasources branch June 29, 2020 21:06

jen-huang mentioned this pull request Jun 29, 2020

[7.x] [Ingest Manager] Support updated package output structure (#69864) #70263

Merged

ruflin mentioned this pull request Jun 30, 2020

[Meta] Planned changes to the manifest format elastic/package-registry#517

Closed

29 tasks

nchaulet mentioned this pull request Jun 30, 2020

[Ingest Manager] Fix agent ack after input format change #70335

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Support updated package output structure #69864

[Ingest Manager] Support updated package output structure #69864

jen-huang commented Jun 24, 2020 •

edited

Loading

elasticmachine commented Jun 25, 2020

paul-tavares commented Jun 26, 2020

jen-huang commented Jun 26, 2020 •

edited

Loading

paul-tavares commented Jun 26, 2020

ruflin Jun 29, 2020

ruflin commented Jun 29, 2020

ruflin left a comment

kibanamachine commented Jun 29, 2020

[Ingest Manager] Support updated package output structure #69864

[Ingest Manager] Support updated package output structure #69864

Conversation

jen-huang commented Jun 24, 2020 • edited Loading

Summary

Example updated agent config yaml

Testing

elasticmachine commented Jun 25, 2020

paul-tavares commented Jun 26, 2020

jen-huang commented Jun 26, 2020 • edited Loading

paul-tavares commented Jun 26, 2020

ruflin Jun 29, 2020

Choose a reason for hiding this comment

ruflin commented Jun 29, 2020

ruflin left a comment

Choose a reason for hiding this comment

kibanamachine commented Jun 29, 2020

💚 Build Succeeded

Build metrics

History

jen-huang commented Jun 24, 2020 •

edited

Loading

jen-huang commented Jun 26, 2020 •

edited

Loading